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LONG-TERM  GOALS 

To  expedite  the  development  of  numerical  weather  prediction  (NWP)  systems,  a  national  partnership 
National  Unified  Operational  Prediction  Capability  (NUOPC)  was  established  between  NOAA  and 
Navy  to  develop  common  software  architecture  for  easy  and  efficient  interoperability.  The  overarching 
goal  of  NUOPC  is  to  accelerate  the  rate  of  improvement  in  the  US  National  enviromnental  prediction 
capability,  focusing  initially  on  the  global  model  enterprise.  Improvements  in  prediction  capability  are 
expected  to  result  in  better  environmental  situation  awareness,  severe  weather  warnings  (hurricanes, 
tornadoes,  snow  storms),  better  cost  avoidance  for  weather  sensitive  industries  (agriculture, 
transportation,  utilities,  and  defense),  and  better  informed  decision  making  for  industry,  defense  and 
the  general  public.  These  goals  will  be  achieved  through  model  development  under  a  common  model 
architecture  and  other  software-related  standards  in  this  project. 

OBJECTIVES 

NUOPC  proposes  to  accelerate  improvement  of  our  national  prediction  capability  in  the  following 
ways:  (1)  Implementing  a  global  atmospheric  multi-model  ensemble  system  designed  to  enhance 
predictive  capability  and  to  provide  probabilistic  prediction  for  severe  weather  events;  (2)  Clearly 
articulating  operational  requirements  and  articulating  a  corresponding  National  research  agenda;  (3) 
Sharing  the  development  efforts  and  promote  collaborations  of  numerical  weather  prediction  (NWP) 
systems  among  the  operational  agencies;  (4)  Accelerating  the  transition  of  new  technology  into  the 
operational  centers;  (5)  Designing  requirements  and  standards  of  future  NWP  systems,  (6) 

Accelerating  the  development  and  transition  of  Navy  global  ensemble  prediction  system  and  its 
participation  in  the  national  multi-model  ensemble  system,  and,  (7)  Implementing  ways  to  enhance 
broad  community  participation  in  addressing  the  National  research  agenda. 
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APPROACH 


This  project  will  support  improvements  and  optimizations  to  the  Earth  System  Modeling  Framework 
(ESMF)  library  and  NUOPC  standard  by  NOAA/ESRL  for  an  efficient  coupling  capability.  ESMF  is 
infrastructure  for  building  and  coupling  weather,  climate,  coastal,  and  other  applications.  The  NUOPC 
Layer  software  is  a  small  set  of  interfaces  that  provide  templates  and  conventions  that  standardize 
ESMF  implementation  and  increase  the  interoperability  of  components.  The  improved  system  will  be 
adopted  and  tested  by  NRL  for  the  Navy  coupled  system.  The  ESMF  development  team  will  provide 
support  for  the  implementation. 

WORK  COMPLETED 

The  most  significant  achievement  during  FY14  was  the  completion  of  a  production  version  and  public 
release  of  the  NUOPC  Layer  software.  Like  ESMF,  the  NUOPC  Layer  is  now  fundamentally 
complete  and  major  changes  to  the  design  and  interface  are  not  anticipated  in  the  future.  However, 
both  ESMF  and  the  NUOPC  Layer  will  require  the  addition  of  capabilities,  options,  and  optimizations 
as  research  and  computational  capabilities  evolve. 

Work  performed  during  FY14  was  in  three  areas:  1)  Design,  implementation,  and  user  support  of  the 
overall  ESMF  architecture,  include  the  machine  representation  (Virtual  Machine)  and  Array  and  Array 
parallel  distribution  representations;  2)  Design,  implementation  and  user  support  of  NUOPC  Layer 
software  and  applications;  and  3)  Design,  implementation,  and  user  support  of  the  ESMF  Field  and 
Field  bundle  classes,  including  a  data  structure  for  conservative  interpolation  called  an  exchange  grid. 
Improvements  to  ESMF  and  NUOPC  software  in  all  three  areas  were  delivered  in  two  releases,  public 
release  ESMF  v6.3.0r  (January  31,  2014)  and  ESMF  patch  release  6.3.0rpl  (July  14,  2014).  Patch 
releases  are  minor  releases  that  address  specific  short-term  needs  or  fixes.  In  the  next  section,  the 
specifics  of  these  ESMF  releases  are  discussed.  Activities  related  to  application  development  and  user 
support  are  described  in  the  section  after  that.  Appendix  A  includes  all  improvements  made  to  the 
ESMF  and  NUOPC  software. 

RESULTS 

1)  ESMF  and  NUOPC  FY14  Releases 

The  ESMF  v6.3.0r  release  concluded  a  development  cycle  that  was  focused  on  supporting  advanced 
grid  remapping  capabilities  for  a  wide  variety  of  applications  in  modeling  and  data  services,  and  in 
producing  a  production-quality  NUOPC  Layer.  ONR  funds  support  the  use  of  ESMF  in  multiple  ways: 
1)  to  create  high-performance,  interoperable  component -based  modeling  systems;  2)  as  a  source  of 
data  communication,  time  management,  metadata  handling,  and  other  libraries;  and  3)  as  a  fast, 
parallel  generator  of  interpolation  weights  from  file  for  many  different  grids  (see 
ESMF_RegridWeightGen,  https://www.earthsystemcog.org/projects/regridweightgen/).  ESMF  can 
also  be  used  in  a  fourth  way,  as  a  Python  grid  remapping  library  (this  is  supported  mainly  by  the 
NOAA  Climate  Program  Office  -  see  ESMPy,  https://www.earthsystemcog.org/projects/esmpy/). 

In  v6.3.0r,  ESMF  regridding  functions  were  expanded  to  include  a  wider  range  of  grids,  interpolation 
methods,  and  control  options.  ESMF  now  supports  during-run  and  offline  grid  remapping  of 
unstructured  grids  composed  of  polygons  with  an  arbitrary  number  of  sides  as  well  as  non-conservative 
regridding  on  either  cell  centers  or  corners  of  unstructured  grids.  The  first  order  conservative 
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interpolation  method  was  extended  to  work  on  3D  grids,  and  nearest-neighbor  interpolation  methods 
were  added.  There  is  also  a  new  option  to  allow  the  user  to  select  between  great  circle  and  Cartesian 
line  paths  when  calculating  interpolation  weights  on  a  sphere  (note  that  this  was  expected  for  FY15 
ONR  tasking  and  was  completed  early).  Grid  files  that  follow  the  Climate  and  Forecast  (CF)  UGRID 
convention  and  the  CF  GRIDSPEC  convention  are  now  both  supported  for  various  object  creation 
methods  and  grid  remapping.  The  ESMF  RegridWeightGen  application  for  calculating  interpolation 
weights  from  file  added  the  option  to  calculate  the  interpolation  and  conservation  error  for  the 
generated  weights  for  a  quick  analysis  of  their  quality. 

ESMF  component  software  has  been  extended  in  a  few  ways.  The  component  interfaces  can  support 
fault-tolerant  interactions  between  components  based  on  a  user-controlled  "timeout"  mechanism,  a 
feature  developed  in  collaboration  with  the  NOAA  Global  Systems  Division.  The  practical  use  cases  of 
this  feature  are  large  ensemble  configurations  where  individual  ensemble  members  may  crash  or 
become  unresponsive  during  the  course  of  a  simulation  run.  Another  new  component-level  feature  is 
the  ability  to  transfer  either  logically  rectangular  Grid  or  unstructured  Mesh  objects  between 
components  that  run  on  different  sets  of  processors.  This  is  useful  when  implementing  coupled  systems 
where  a  component  needs  to  access  fields  of  another  component  on  that  component's  native  grid. 

The  NUOPC  Layer  software  was  ported  to  all  ESMF  supported  platforms  and  thoroughly  regression 
tested.  A  run-time  tool  called  a  “Component  Explorer”  was  added,  which  can  be  used  to  analyze  a 
component’s  compliance  status.  Other  additions  include  support  for  resolving  data  dependencies 
during  initialization  for  components  that  are  running  concurrently,  and  defining  a  standard  way  to 
combine  the  build  systems  of  different  components  when  coupled  together. 

The  NUOPC  Layer  improvements  and  fault  tolerance  mechanisms  are  implemented  in  the  v6.3.0r 
release.  There  are  continued  implementations  of  exchange  grids  (X Grids).  These  data  objects  conserve 
flux  quantities  as  they  pass  through  the  interface  between  domains  (Balaji  et  al.  2007).  The 
implementation  now  allows  combinations  of  logically  rectangular  Grids  and  unstructured  Meshes  to  be 
supplied  on  either  side  of  the  XGrid.  Weights  generated  by  regridding  through  the  XGrid  can  be 
renonnalized  by  user-supplied  source  or  destination  area,  and  area  and  centroid  information  can  be 
retrieved. 

In  public  patch  release  ESMF  v6.3.0rpl,  the  ESMF  grid  remapping  utilities  were  extended  as  a  result 
of  user  feedback  to  offer  destination  fraction  normalization  for  the  conservative  interpolation  scheme 
in  addition  to  the  regular  destination  area  normalization.  Also,  the  portability  of  the  library  was 
enhanced  in  several  areas:  ESMF  can  now  be  built  with  native  support  for  the  Intel  MIC  architecture, 
the  Apple  Clang/LLVM  C++  compiler  is  supported  on  both  Linux  and  Darwin,  and  ESMF's 
dependency  on  the  NetCDF  C++  interface  has  been  removed  to  reduce  the  number  of  build 
dependencies. 

2)  Applications  and  User  Support 

The  ESMF  and  NUOPC  team  interacted  with  NRL  developers  on  a  number  of  support  issues  and 
feature  requests  in  FY14.  These  interactions  included  the  following: 

The  ESMF  grid  remapping  lead  (Oehmke)  organized  a  call  among  ESMF  development  team  and  NRL 
scientists  to  discuss  how  best  to  handle  unmapped  points  during  grid  remapping.  This  ability  to 
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extrapolate  to  points  that  lie  outside  the  source  grid  is  currently  scheduled  for  delivery  in  release  7.1, 
during  FY15. 

Other  interactions  involved  helping  NRL  scientists  to  use  high  order  patch  interpolation  for  grid 
remapping  and  to  identify  a  suitable  pole  handling  method.  Another  problem  solved  was  on  the 
NetCDF3  file  format  when  the  interpolation  weights  were  in.  The  weight  file  was  so  large  that  a 
NetCDF4  format  would  solve  the  problem.  Other  problems  solved  included  a  mismatch  in  coordinate 
systems,  and  the  grid  remapping.  One  of  the  improvements  in  6.3.0rpl,  the  removal  of  the  dependency 
on  the  NetCDF  C++  interface,  addressed  the  porting  issue  and  James  was  able  to  get  ESMF  built  on 
the  upgraded  system  by  updating  to  the  new  ESMF  version. 

One  interaction  resulted  in  changing  ESMF  so  that  it  reports  a  non-zero  error  code  to  MPI  Abort  calls, 
which  allows  scripts  that  are  calling  the  program  to  tell  that  an  error  occurred.  NRL  scientists  also 
raised  the  possibility  of  ESMF  providing  a  transpose  of  the  regrid/redistribution  operations  for  data 
assimilation.  Other  ESMF  team  actions  included  fixing  a  small  issue  with  CPPFLAGS  that  Tim 
identified,  iterating  with  him  on  the  best  way  to  support  the  IBM  iDataPlex  build,  and  incorporating 
contributed  code  from  Tim  into  the  ESMF  build  which  makes  it  easier  for  ESMF  to  find  paths  for 
NetCDF4  and  its  associated  HDF  dependency. 

IMPACT/APPLICATIONS 

Interoperability  among  national  operational  and  development  agencies  will  lead  to  accelerated 
advancement  of  the  environmental  numerical  prediction,  better  sharing  of  computing  resources  and 
more  reliable  operational  backup.  The  Guidance  Enhancement  system  will  enable  Navy  production 
centers  to  produce  the  best  guidance  possible  to  support  forecasts  and  associated  decisions.  The 
products  will  support  improved  high  wind  and  seas  warnings,  and  in  addition,  the  enhanced  uncertainty 
products  will  provide  improved  guidance  for  Level  3  forecasters  in  the  Watch  Floor  of  the  Future 
construct. 

RELATED  PROJECTS 

6.4  ESPC  Coupled  Global  system,  6.4  NAVGEM. 
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Appendix  A:  ESMF  /  NUOPC  Improvements  for  FY14  Releases 


ESMF  6. 3. Or  (January  2014) 

•  The  NUOPC  Layer  contained  in  this  release  has  been  improved  in  the  following  specific 
technical  areas: 

•  The  NUOPC  Component  Explorer  was  added.  This  is  a  run-time  tool  that  can  be  used  to 
analyze  the  compliance  status  of  a  Component. 

•  Standard  Component  dependencies  have  been  formalized  and  documented,  making 
Components  more  interoperable  on  the  build  level. 

•  The  resolution  of  data  dependencies  during  initialization  was  extended  to  cover  components 
that  run  on  different  petLists. 

•  The  transfer  of  Grid  and  Mesh  objects  between  Components  is  now  supported  during 
initialization. 

•  The  tools  that  allow  Components  to  be  made  available  as  web-based  services  now  provide  the 
capability  to  use  a  job  scheduler  to  manage  the  execution  and  resource  allocation  for  the 
service. 

•  The  capability  to  create  logically  rectangular  Grid  objects  from  NetCDF  files  in  SCRIP  format 
was  added  to  ESMPy,  the  ESMF  Python  interface.  Documentation  and  downloads  are  available 
on  the  ESMPy  web  page. 

•  Distributed  objects  Fields,  Arrays,  Grids,  and  Meshes  created  in  one  Component  can  now  be 
transferred  to  another  Component,  even  when  the  Components  are  defined  on  different 
petLists. 

•  The  "—check"  option  was  added  to  the  ESMF  RegridWeightGen  application  to  calculate  the 
interpolation  and  conservation  error  from  regridding  with  the  weights  in  the  output  file.  The 
error  is  calculated  using  an  analytical  test  field. 

•  The  ESMF_RegridWeightGen()  interface  was  overloaded  to  allow  regridding  between  grids 
with  user  specified  element  and  optional  nodal  distribution.  The  grids  must  be  supplied  in  the 
SCRIP  format. 

•  The  ESMF_FieldRegridStore()  call  has  been  extended  to  work  on  Fields  built  on  Meshes 
containing  elements  with  more  than  four  sides  (e.g.  pentagons  and  hexagons). 

•  The  ESMF_FieldRegridStore()  call  has  been  extended  so  that  non-conservative  interpolation 
methods  (bilinear,  patch,  and  nearest-neighbor)  can  be  performed  on  Fields  built  on  the  Mesh 
element  location  (E SMF_ME S HFO C_ELEMENT ) . 

•  The  ESMF_FieldRegridStore()  call  now  allows  the  user  to  choose  between  great  circle  or 
Cartesian  paths  for  the  line  between  two  points  on  a  sphere  during  the  calculation  of  bilinear 
interpolation  weights. 

•  The  ESMF_FieldEmptySet()  method  now  allows  the  Grid,  Mesh,  LocStream,  or  XGrid  object 
to  be  replaced  in  a  partially  completed  Field. 

•  The  ESMF_MeshCreate()  interfaces  have  been  extended  to  allow  the  user  to  specify  a 
coordinate  system.  This  information  is  used  by  the  ESMF_FieldRegridStore()  call  to 
automatically  determine  the  geometry  of  the  source  and  destination  Meshes. 

•  The  ESMF_MeshCreate()  and  ESMF_MeshAddElements()  methods  have  been  extended  to 
allow  the  user  to  create  a  Mesh  containing  elements  with  more  than  four  sides  (e.g.  pentagons 
and  hexagons). 
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•  The  ESMF_MeshCreate()  and  ESMF_MeshAddElements()  methods  have  been  extended  to 
allow  the  user  to  optionally  set  element  center  coordinates.  This  information  is  used  when 
doing  non-conservative  regridding  on  Fields  built  on  the  Mesh  element  location 
(ESMF_MESHLOC_ELEMENT). 

•  The  ESMF_MeshCreate()  which  allows  the  user  to  to  create  a  copy  of  an  existing  Mesh,  but 
with  a  different  distribution,  now  allows  the  user  to  optionally  omit  the  node  distribution. 

•  The  ESMF_MeshCreate()  method  has  been  extended  to  allow  generation  of  the  nodal  mask 
when  reading  from  a  UGRID  file. 

•  The  ESMF_MeshCreate()  method  has  been  updated  to  support  the  new  UGRID  version  0.9.0 
conventions. 

•  The  ESMF_MeshCreate()  method  has  been  extended  to  allow  Meshes  to  be  created  from  a  grid 
file  with  user  specified  element  and  optional  nodal  distribution. 

•  The  ESMF_GridCreate()  interface  was  overloaded  to  allow  Grids  to  be  created  from  a  grid  file 
with  user  specified  distribution. 

•  The  ESMF  AttPack  type  was  introduced,  allowing  a  user  to  retrieve  a  handle  to  an  ESMF 
Attribute  package.  This  is  a  more  compact  way  of  working  with  Attribute  packages. 

•  The  ESMF_UtilIOGetCWD()  interface  was  added  to  allow  retrieval  of  the  absolute  path  and 
name  of  the  current  working  directory. 

•  A  new  parallel  application  ESMF_Scrip2Unstruct  was  added  to  convert  a  SCRIP  format  grid 
file  to  an  unstructured  ESMF  format  or  a  UGRID  format  grid  file.  This  allows  the  user  to 
execute  the  potentially  expensive  file  format  conversion  operation  once  instead  of  every  time 
the  ESMF  RegridWeightGen  application  is  called.  The  SCRIP  file  can  be  either  unstructured 
or  logically  rectangular.  The  application  has  an  option  to  generate  a  dual  mesh,  which  is  needed 
to  support  non-conservative  regridding  on  cell  centers,  regridding,  and  numerical  results  of 
some  specific  test  cases. 


ESMF  6.3.0rpl  (July  2014) 

•  An  optional  command  line  argument  no  log"  was  added  to  the  ESMF  RegridWeightGen 
application.  This  allows  the  user  to  turn  off  the  ESMF  log  files  that  are  written  by  default. 

•  The  ESMF  regrid  weight  generation  system  now  allows  the  user  to  choose  between  two 
normalization  types  when  generating  conservative  interpolation  weights:  destination  area 
normalization  (default)  and  destination  fraction  normalization. 

•  The  ESMF  Python  interface  (ESMPy)  is  now  part  of  the  ESMF  source  distribution.  The  build 
and  installation  process  for  both  ESMF  and  ESMPy  remain  separate  in  this  release. 

•  When  building  ESMF  with  NetCDF  support  enabled,  the  NetCDF  C++  interface  is  no  longer 
required. 

•  New  build  configurations  were  added  to  support  the  Apple  Clang/LLVM  C++  compiler  on  both 
Linux  and  Darwin  systems. 

•  Support  was  added  to  compile  the  ESMF  library  for  the  Intel  MIC  architecture  under  Linux. 
This  allows  ESMF  applications  to  execute  natively  on  the  Intel  Xeon  Phi  accelerator  hardware. 
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